Integrating Gaze, image analysis, and body tracking: Foothold selection during locomotion.

Relatively little is known about the way vision is use to guide locomo-tion in the natural world. What visual features are used to choose paths in natural complex terrain? How do walkers trade off different costs such as getting to the goal, minimizing energy, and satisfying stability constraints? To answer these questions, it is necessary to monitor not only the eyes and the body, but also to represent the three dimensional structure of the terrain. We used photogrammetry techniques to do this, and found substantial regularities in the choice of paths. Walkers avoid paths that involve changes in height and choose more circuitous and flatter paths. This stable tradeoff is related to the walker's leg length and reflects both energetic and stability constraints. Gaze data and path choices suggest that subjects take into account the terrain approximately 5 steps ahead, and so are planning routes as well as particular footplants. Such planning ahead allows the minimization of energetic costs. Thus locomotor behavior in natural environments is controlled by decision mechanisms that attempt to optimize for multiple factors in the context of well-calibrated sensory and motor internal models.

Retinal motion statistics during natural locomotion.

Walking through an environment generates retinal motion, which humans rely on to perform a variety of visual tasks. Retinal motion patterns are determined by an interconnected set of factors, including gaze location, gaze stabilization, the structure of the environment, and the walker's goals. The characteristics of these motion signals have important consequences for neural organization and behavior. However, to date, there are no empirical in situ measurements of how combined eye and body movements interact with real 3D environments to shape the statistics of retinal motion signals. Here, we collect measurements of the eyes, the body, and the 3D environment during locomotion. We describe properties of the resulting retinal motion patterns. We explain how these patterns are shaped by gaze location in the world, as well as by behavior, and how they may provide a template for the way motion sensitivity and receptive field properties vary across the visual field.

Tasks and their role in visual neuroscience.

Vision is widely used as a model system to gain insights into how sensory inputs are processed and interpreted by the brain. Historically, careful quantification and control of visual stimuli have served as the backbone of visual neuroscience. There has been less emphasis, however, on how an observer's task influences the processing of sensory inputs. Motivated by diverse observations of task-dependent activity in the visual system, we propose a framework for thinking about tasks, their role in sensory processing, and how we might formally incorporate tasks into our models of vision.

From Puzzle to Progress: How Engaging With Neurodiversity Can Improve Cognitive Science.

In cognitive science, there is a tacit norm that phenomena such as cultural variation or synaesthesia are worthy examples of cognitive diversity that contribute to a better understanding of cognition, but that other forms of cognitive diversity (e.g., autism, attention deficit hyperactivity disorder/ADHD, and dyslexia) are primarily interesting only as examples of deficit, dysfunction, or impairment. This status quo is dehumanizing and holds back much-needed research. In contrast, the neurodiversity paradigm argues that such experiences are not necessarily deficits but rather are natural reflections of biodiversity. Here, we propose that neurodiversity is an important topic for future research in cognitive science. We discuss why cognitive science has thus far failed to engage with neurodiversity, why this gap presents both ethical and scientific challenges for the field, and, crucially, why cognitive science will produce better theories of human cognition if the field engages with neurodiversity in the same way that it values other forms of cognitive diversity. Doing so will not only empower marginalized researchers but will also present an opportunity for cognitive science to benefit from the unique contributions of neurodivergent researchers and communities.

Motion vision: Fish swimming to see.

Self-motion generates optic flow, a visual motion signal used by many organisms for navigation and self-stabilization. A new study quantitatively demonstrates how environmental structure and current behavioral state explain the spatial biases observed in zebrafish optomotor responses.

Stereo slant discrimination of planar 3D surfaces: Frontoparallel versus planar matching.

Binocular stereo cues are important for discriminating 3D surface orientation, especially at near distances. We devised a single-interval task where observers discriminated the slant of a densely textured planar test surface relative to a textured planar surround reference surface. Although surfaces were rendered with correct perspective, the stimuli were designed so that the binocular cues dominated performance. Slant discrimination performance was measured as a function of the reference slant and the level of uncorrelated white noise added to the test-plane images in the left and right eyes. We compared human performance with an approximate ideal observer (planar matching [PM]) and two subideal observers. The PM observer uses the image in one eye and back projection to predict a test image in the other eye for all possible slants, tilts, and distances. The estimated slant, tilt, and distance are determined by the prediction that most closely matches the measured image in the other eye. The first subideal observer (local planar matching [LPM]) applies PM over local neighborhoods and then pools estimates across the test plane. The second suboptimal observer (local frontoparallel matching [LFM]) uses only location disparity. We find that the ideal observer (PM) and the first subideal observer (LPM) outperforms the second subideal observer (LFM), demonstrating the additional benefit of pattern disparities. We also find that all three model observers can account for human performance, if two free parameters are included: a fixed small level of internal estimation noise, and a fixed overall efficiency scalar on slant discriminability.

Retinal optic flow during natural locomotion.

We examine the structure of the visual motion projected on the retina during natural locomotion in real world environments. Bipedal gait generates a complex, rhythmic pattern of head translation and rotation in space, so without gaze stabilization mechanisms such as the vestibular-ocular-reflex (VOR) a walker's visually specified heading would vary dramatically throughout the gait cycle. The act of fixation on stable points in the environment nulls image motion at the fovea, resulting in stable patterns of outflow on the retinae centered on the point of fixation. These outflowing patterns retain a higher order structure that is informative about the stabilized trajectory of the eye through space. We measure this structure by applying the curl and divergence operations on the retinal flow velocity vector fields and found features that may be valuable for the control of locomotion. In particular, the sign and magnitude of foveal curl in retinal flow specifies the body's trajectory relative to the gaze point, while the point of maximum divergence in the retinal flow field specifies the walker's instantaneous overground velocity/momentum vector in retinotopic coordinates. Assuming that walkers can determine the body position relative to gaze direction, these time-varying retinotopic cues for the body's momentum could provide a visual control signal for locomotion over complex terrain. In contrast, the temporal variation of the eye-movement-free, head-centered flow fields is large enough to be problematic for use in steering towards a goal. Consideration of optic flow in the context of real-world locomotion therefore suggests a re-evaluation of the role of optic flow in the control of action during natural behavior.

Binocular vision and the control of foot placement during walking in natural terrain.

Coordination between visual and motor processes is critical for the selection of stable footholds when walking in uneven terrains. While recent work (Matthis et al. in Curr Biol 8(28):1224-1233, 2018) demonstrates a tight link between gaze (visual) and gait (motor), it remains unclear which aspects of visual information play a role in this visuomotor control loop, and how the loss of this information affects that relationship. Here we examine the role of binocular information in the visuomotor control of walking over complex terrain. We recorded eye and body movements while normally-sighted participants walked over terrains of varying difficulty, with intact vision or with vision in one eye blurred to disrupt binocular vision. Gaze strategy was highly sensitive to the complexity of the terrain, with more fixations dedicated to foothold selection as the terrain became more difficult. The primary effect of increased sensory uncertainty due to disrupted binocular vision was a small bias in gaze towards closer footholds, indicating greater pressure on the visuomotor control process. Participants with binocular vision losses due to developmental disorders (i.e., amblyopia, strabismus), who have had the opportunity to develop alternative strategies, also biased their gaze towards closer footholds. Across all participants, we observed a relationship between an individual's typical level of binocular visual function and the degree to which gaze is shifted toward the body. Thus the gaze-gait relationship is sensitive to the level of sensory uncertainty, and deficits in binocular visual function (whether transient or long-standing) have systematic effects on gaze strategy in complex terrains. We conclude that binocular vision provides useful information for locating footholds during locomotion. Furthermore, we have demonstrated that combined eye/body tracking in natural environments can be used to provide a more detailed understanding of the impact of a type of vision loss on the visuomotor control process of walking, a vital everyday task.

Binocular viewing geometry shapes the neural representation of the dynamic three-dimensional environment.

Sensory signals give rise to patterns of neural activity, which the brain uses to infer properties of the environment. For the visual system, considerable work has focused on the representation of frontoparallel stimulus features and binocular disparities. However, inferring the properties of the physical environment from retinal stimulation is a distinct and more challenging computational problem-this is what the brain must actually accomplish to support perception and action. Here we develop a computational model that incorporates projective geometry, mapping the three-dimensional (3D) environment onto the two retinae. We demonstrate that this mapping fundamentally shapes the tuning of cortical neurons and corresponding aspects of perception. For 3D motion, the model explains the strikingly non-canonical tuning present in existing electrophysiological data and distinctive patterns of perceptual errors evident in human behavior. Decoding the world from cortical activity is strongly affected by the geometry that links the environment to the sensory epithelium.

Beyond Trial-Based Paradigms: Continuous Behavior, Ongoing Neural Activity, and Natural Stimuli.

The vast majority of experiments examining perception and behavior are conducted using experimental paradigms that adhere to a rigid trial structure: each trial consists of a brief and discrete series of events and is regarded as independent from all other trials. The assumptions underlying this structure ignore the reality that natural behavior is rarely discrete, brain activity follows multiple time courses that do not necessarily conform to the trial structure, and the natural environment has statistical structure and dynamics that exhibit long-range temporal correlation. Modern advances in statistical modeling and analysis offer tools that make it feasible for experiments to move beyond rigid independent and identically distributed trial structures. Here we review literature that serves as evidence for the feasibility and advantages of moving beyond trial-based paradigms to understand the neural basis of perception and cognition. Furthermore, we propose a synthesis of these efforts, integrating the characterization of natural stimulus properties with measurements of continuous neural activity and behavioral outputs within the framework of sensory-cognitive-motor loops. Such a framework provides a basis for the study of natural statistics, naturalistic tasks, and/or slow fluctuations in brain activity, which should provide starting points for important generalizations of analytical tools in neuroscience and subsequent progress in understanding the neural basis of perception and cognition.

Dynamic mechanisms of visually guided 3D motion tracking.

The continuous perception of motion-through-depth is critical for both navigation and interacting with objects in a dynamic three-dimensional (3D) world. Here we used 3D tracking to simultaneously assess the perception of motion in all directions, facilitating comparisons of responses to motion-through-depth to frontoparallel motion. Observers manually tracked a stereoscopic target as it moved in a 3D Brownian random walk. We found that continuous tracking of motion-through-depth was selectively impaired, showing different spatiotemporal properties compared with frontoparallel motion tracking. Two separate factors were found to contribute to this selective impairment. The first is the geometric constraint that motion-through-depth yields much smaller retinal projections than frontoparallel motion, given the same object speed in the 3D environment. The second factor is the sluggish nature of disparity processing, which is present even for frontoparallel motion tracking of a disparity-defined stimulus. Thus, despite the ecological importance of reacting to approaching objects, both the geometry of 3D vision and the nature of disparity processing result in considerable impairments for tracking motion-through-depth using binocular cues.NEW & NOTEWORTHY We characterize motion perception continuously in all directions using an ecologically relevant, manual target tracking paradigm we recently developed. This approach reveals a selective impairment to the perception of motion-through-depth. Geometric considerations demonstrate that this impairment is not consistent with previously observed spatial deficits (e.g., stereomotion suppression). However, results from an examination of disparity processing are consistent with the longer latencies observed in discrete, trial-based measurements of the perception of motion-through-depth.

Speed discrimination in the far monocular periphery: A relative advantage for interocular comparisons consistent with self-motion.

Some animals with lateral eyes (such as bees) control their navigation through the 3D world using velocity differences between the two eyes. Other animals with frontal eyes (such as primates, including humans) can perceive 3D motion based on the different velocities that a moving object projects upon the two retinae. Although one type of 3D motion perception involves a comparison between velocities from vastly different (monocular) portions of the visual field, and the other involves a comparison within overlapping (binocular) portions of the visual field, both compare velocities across the two eyes. Here we asked whether human interocular velocity comparisons, typically studied in the context of binocularly overlapping vision, operate in the far lateral (and hence, monocular) periphery and, if so, whether these comparisons were accordant with conventional interocular motion processing. We found that speed discrimination was indeed better between the two eyes' monocular visual fields, as compared to within a single eye's (monocular) visual field, but only when the velocities were consistent with commonly encountered motion. This intriguing finding suggests that mechanisms sensitive to relative motion information on opposite sides of an animal may have been retained, or at some point independently achieved, as the eyes became frontal in some animals.

Continuous psychophysics: Target-tracking to measure visual sensitivity.

We introduce a novel framework for estimating visual sensitivity using a continuous target-tracking task in concert with a dynamic internal model of human visual performance. Observers used a mouse cursor to track the center of a two-dimensional Gaussian luminance blob as it moved in a random walk in a field of dynamic additive Gaussian luminance noise. To estimate visual sensitivity, we fit a Kalman filter model to the human tracking data under the assumption that humans behave as Bayesian ideal observers. Such observers optimally combine prior information with noisy observations to produce an estimate of target position at each time step. We found that estimates of human sensory noise obtained from the Kalman filter fit were highly correlated with traditional psychophysical measures of human sensitivity (R2 > 97%). Because each frame of the tracking task is effectively a "minitrial," this technique reduces the amount of time required to assess sensitivity compared with traditional psychophysics. Furthermore, because the task is fast, easy, and fun, it could be used to assess children, certain clinical patients, and other populations that may get impatient with traditional psychophysics. Importantly, the modeling framework provides estimates of decision variable variance that are directly comparable with those obtained from traditional psychophysics. Further, we show that easily computed summary statistics of the tracking data can also accurately predict relative sensitivity (i.e., traditional sensitivity to within a scale factor).